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Abstract 

Using data collected from peer reviews for Open Textbook Library titles, this paper explores questions about 
rating the quality of open textbooks. The five research questions addressed the relationship between 
textbook and reviewer characteristics and ratings. Although reviewers gave textbooks high ratings 
generally, reviewers identified differences in quality according to criteria and discipline. Physics and 
chemistry textbooks earned significantly lower ratings than other textbook types. Ratings were not 
significantly associated with reviewers’ status and experience, but were associated with reviewers’ place of 
residence. We discuss the implications for OER efficacy studies and other research. 
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Introduction 

For several years, the cost of higher education has been increasing at an alarming rate. The U.S. Government 
Accountability Office (2013) reported that from 2002 to 2012, tuition and fees increased by 89% and new 
textbook prices increased 82%, while overall consumer prices increased 28%. Each year over 1 million 
college students graduate with student loan debt averaging about $30,000 per student, which has very 
serious implications (Cochrane & Reed, 2015). Part of that debt can be attributed to textbook costs. Faced 
with prohibitive costs, some students are enrolling in fewer courses than they otherwise would, taking on 
more debt, or choosing not to buy required texts (Hilton, Gaudet, Clark, Robinson, and Wiley, 2013; Senack, 
2014). 

One promising development helping to address burgeoning college costs is the growing availability of open 
textbooks. Open textbooks are a type of Open Educational Resource (OER), with an open license that may 
allow copying, distribution, modification, and in some cases, sale (Bissell, 2009). Most open textbooks are 
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available free online, often with a low-cost print version. Potential benefits of using open textbooks include 
significant cost savings to students and increased flexibility for instructors (Hilton et al., 2013; Senack, 
2015). 

Although open textbooks offer some substantial cost-advantages, they represent only a small fraction of the 
textbooks adopted by faculty members. One reason for this low adoption is that most professors do not 
know that open textbooks exist, much less where to find them. In a nationally representative survey of 
college faculty, Allen and Seaman (2014) found that two-thirds of the 2,144 respondents were not aware of 
OER. Even instructors who are aware may avoid open textbooks because they question the quality of freely 
available resources, especially when such sources aren’t connected to a reputable publishing house, 
institution, or author (Pitt, 2015). College instructors who have their students’ best interest in mind are not 
willing to replace a high-quality, high-cost textbook with a low-quality, low-cost alternative, and 
determining the quality of a non-traditional text may take more time and effort than instructors prefer to 
spend. 

To help instructors determine the quality of open textbooks, the Open Textbook Library, housed in the 
Center for Open Education at the University of Minnesota, has been aggregating peer reviews of these 
textbooks. Reviews are for faculty, by faculty; “The project pays faculty members to review open textbooks, 
as does a similar effort at the University of British Columbia (Canada), and the two initiatives share those 
peer reviews” (Center for Education Attainment and Innovation, 2015, p.2). To be included in the library, 
textbooks “[mjust be currently in use at multiple higher education institutions, or affiliated with a higher 
education institution, scholarly society, or professional organization” (Center for Open Education, n.d., 
para. 1). In this paper, we examine data collected from peer reviews for Open Textbook Library titles to 
discuss questions about rating the quality of OER. 


Review of Literature 

Several recent studies provide insight into faculty perceptions of open textbooks. As of this writing, all 
published perception studies report positive perceptions of open textbooks. Petrides, Jimes, Middleton- 
Detzner, Walling, and Weiss (2011) reported perceptions of OER by faculty and students who had used an 
open statistics textbook. The 31 instructors interviewed cited cost, content quality, and ease of use as the 
main factors that influenced them to adopt open textbooks. For the 45 students, advantages of OER 
included cost, portability, and ease of use. Both instructors and students said that the open textbook had 
positive impacts on teaching and learning. In a similar study focusing on a single open textbook, Lindshield 
and Adhikari (2013) examined online and campus students’ perceptions of an open textbook used in 
Human Nutrition courses. While online students used the open textbook more frequently and rated it more 
highly than campus students, both campus and online students reported high levels of satisfaction and 
preferred using the online text to buying a traditional textbook. 

Unlike the previous two studies, most research articles on OER involve multiple open resources. In one 
such study, Bliss, Hilton, Wiley, and Thanos (2013) reported results from a survey of 11 instructors and 132 
students using OER at eight community colleges. A large majority of respondents reported positive 
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perceptions of OER. Ten of the eleven instructors stated their students were as or more prepared than 
students using traditional texts, and 97% of students said that the open textbooks were as good as or better 
than traditional textbooks. Building on the previous study, Bliss, Robinson, Hilton, and Wiley (2013), 
surveyed an additional 58 teachers and 490 students about the cost, outcomes, usage and quality of OER 
used in their college courses. As with the previous study, a large majority of respondents rated OER as equal 
or higher in quality compared to traditional textbooks. The respondents who rated OER quality lower than 
that of traditional textbooks (11% of instructors and 6 % of students) cited poor text quality and technology 
problems as the reasons for their ratings. 

Hilton, Gaudet, Clark, Robinson, and Wiley (2013) examined the impact, in terms of cost savings, student 
retention and success, and user perceptions of adopting open resources in mathematics classes at Scottsdale 
Community College (SCC). Using data from 2,043 students taking five different courses, researchers 
determined that by using OER, students saved a significant amount of money without suffering adverse 
effects in terms of educational outcomes. In surveys, students and faculty reported mostly favorable 
perceptions of the OER materials used in the courses. 

Feldstein et al. (2012) reported findings from a year-long pilot study in which 991 students taking courses 
in the Virginia State University School of Business used free online textbooks and other resources provided 
by Flat World Knowledge. Of the 315 students who completed the semester-end survey, 95% agreed that 
the OER course materials were easy to use, and most respondents preferred the digital content to traditional 
textbooks. Students in the courses that used open textbooks earned higher grades than students in similar 
courses that used traditional textbooks. 

All of the above studies include student perceptions; several studies focus only on instructor perceptions. 
In their 2014 article, Allen and Seaman (2014) reported that among respondents aware of OER, 74% rated 
the quality of OER materials as similar or superior to that of traditional resources. About 85% rated the 
efficacy of OER materials as similar or superior to the efficacy of traditional materials. Kelly’s 2014 study 
examined educator perceptions of OER and the impact of perception on adoption and use. Survey 
participants included 128 educators from higher education, K-12 schools, and workplace training. Kelly 
found that individuals with high self-efficacy were more likely to perceive OER as easy to use, and that ease 
of use correlated with perceived usefulness. 

As part of a collaborative project between the BCcampus-led Open Textbook Project and OER Hub, 
researchers surveyed post-secondary educators interested in OER to examine awareness, usage, barriers, 
outcomes, and perceptions (Jhangiani, Pitt, Hendricks, Key, & Lalonde, 2016). Of the 78 respondents, 77% 
had used OER. Most respondents rated OER quality as comparable or superior to that of traditional 
materials; educators who had adopted OER rated the quality significantly higher than educators who had 
not. Respondents reported that the top two barriers to using OER were finding relevant and high quality 
OER. Faculty at research-intensive universities reported significantly lower barriers to finding high-quality 
OER than did faculty at teaching-intensive universities or colleges/institutes. 

Pitt (2015) reported results of two surveys of 126 educators who had used open textbooks that were 
published by OpenStax. In relation to the issue of quality, almost 75% of the 2013 respondents and 95.7% 
of 2014/2015 respondents reported that they would be more likely to choose OER produced by a trusted 
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author than OER produced by a lesser known or trusted entity. Pitt concludes that “the role of educators in 
promoting and giving credibility to OER, such as open textbooks, cannot be underestimated” (2015, p.141). 

To date, concerning OER quality have typically been surveys of students and/or faculty members that have 
used OER in their classroom. One weakness with this approach is that students may be more likely to report 
based on convenience, rather than true efficacy (Kvavik, 2005). In addition, the faculty members may have 
been biased in terms of reviewing materials that in many instances they helped to create or select. In the 
present study, we will address this limitation by looking at a different aspects of reviews of open textbook 
quality. Rather than use survey results, we will us data from the peer reviews facilitated by the Open 
Textbook Library. These faculty reviewers had not adopted the resource at the time of their review. 
Therefore, their reviews may be somewhat less biased. Moreover, beyond reporting descriptive values of 
quality, we considered evaluations in terms of reviewers’ characteristics (tenure status, job track, location), 
and textbook content areas. Our specific research questions will be discussed below. 


Method 

The Open Text Network housed in the College of Education and Human Development at the University of 
Minnesota (open.umn.edu) acquires and disseminates OER written for higher education. By January 2016 
it had catalogued 210 textbooks in the Open Textbook Library. Faculty from the US and Canada were invited 
to review and evaluate the quality of the textbooks. 

Faculty reviewers evaluated 121 of the textbooks across 10 qualities on a 5-point Likert scale anchored with 
“1” as low and “5” as high. The 10 criteria were Comprehensiveness, Accuracy, Relevance/Longevity, Clarity, 
Consistency, Modularity, Organization, Interface, Grammar, and Cultural Relevance. Researchers summed 
the evaluations across all 10 qualities and divided by 10 to estimate an Overall Quality value. Included in 
this study were 416 reviews of the 121 textbooks. Of the 416 reviews, 377 were by faculty at the University 
of Minnesota; 44 were by faculty of British Columbia College in Vancouver, Canada. Some textbooks 
received multiple reviews with the number of reviews per textbook ranging from 1 to 21. 

Reviewers were sorted into types for analysis purposes. Reviewers could be considered as Tenure Track or 
Non-tenure Track. They were further sorted into four groups: Adjunct, Tenure Track Assistant Professors, 
Tenured Associate or Full Professors and Other (such as student life advisors that teach courses). Reviewers 
were also sorted by US and Canada origin. 

Textbooks were sorted into nine discipline types: Math/Statistics, Biology/Physiology, Physics/Chemistry, 
Social Science, English/Humanities/Communications, Computer Science/Information Systems, 
Business/Economics, Law, and Other. 

Researchers sought answers to five research questions: 

Question 1: What are the average ratings of each quality criterion and overall quality? 

Question 2: Do Tenure Track and Non-tenure Track reviewers rate the textbooks differently? 
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Question 3: Do the four subgroups of reviewers rate the textbooks differently? 

Question 4: Do the US and Canadian reviewers rate the textbooks differently? 

Question 5: Do the nine textbook types differ across the quality criteria or overall quality? 

Because the data were ordinal Likert scales and the distributions were significantly skewed (as seen in Table 
1), all inferential tests were completed using non-parametric Kruskal-Wallis tests of differences. Although 
the evaluations were nested with repeated measures, the data were not robust enough to use multi-level 
modeling. The results should be interpreted cautiously, with an eye toward general trends more than solid 
inferential tests of null hypotheses. 


Results 

The distributions of the rating across the 10 quality criteria and overall quality are displayed in Table 1. In 
general, the median ratings were high (4 & 5). The means were high with reasonable standard deviations 
and all the distributions were significantly negatively skewed. 

Table 1 

Descriptive Statistics of Criteria Variables 


Criteria 

Mean 

Median 

Standard 

deviation 

Skew Standard 

error of 
skew 

f-test 

of 

skew 

Comprehensiveness 

4-23 

4.00 

0.816 

-0.95 

0.12 

-7.90 

Accuracy 

4.41 

5.00 

0.740 

-1.15 

0.12 

- 9-54 

Relevance/longevity 

4.28 

4.00 

0.803 

-1.07 

0.12 

-8.88 

Clarity 

4-43 

5.00 

0.800 

-1.48 

0.12 

-12.37 

Consistency 

4-57 

5.00 

0.723 

-1.97 

0.12 

-16.40 

Modularity 

4-49 

5.00 

0-759 

-1.38 

0.12 

-11.49 

Organization/flow 

4.28 

4.00 

0.849 

-1.16 

0.12 

-9.70 
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Interface 

4.04 

4.00 

0-975 

-0.82 

0.12 

-6.84 

Grammar 

4.71 

5.00 

0.605 

-2.41 

0.12 

-20.11 

Cultural relevance 

4.20 

4.00 

0.968 

-1-33 

0.12 

-9-45 

Overall rating 

4-36 

4-50 

0.474 

-1.28 

0.12 

-10.65 


Table 2 illustrates the results associated with all inferential tests. Other tables will present the descriptive 
statistics across all criteria. As seen in Table 3, all reviewers generally rated the textbooks favorably across 
the 10 criteria. There were no significant differences between the Tenure Track and Non-tenure Track 
reviewers on any of the 10 quality criteria or overall quality score (Table 2). The only significant difference 
among the four subgroups of reviewers on quality criteria or overall quality score was observed between the 
Other reviewers and the Adjunct Faculty reviewers in terms of their assessment of modularity (Table 2). 
The Other reviewers judged the modularity of the books to be lower than Adjunct Faculty reviewers. 

Table 2 


Summary of Kruskal-Wallis Tests of Differences Across Groups in Terms of 10 Criteria 


Criteria 

Reviewer 

status 

(Table 2) 

p-value 

Adjunct 
v tenure 

track 

(Table 

2) 

p-value 

Associate 
v assistant 
(Table 2) 

BCCv 
UMN 
(Table 
p-value 3) 

p-value 

Text 

type 

(Table 

4) 

p-value 

Comprehensiveness 2.105 

0.551 

0.106 

0-745 

0.026 

0.872 

12.03 

0.001* 

7.022 

0-534 

Accuracy 

6.769 

0.08 

0.003 

0-955 

5-058 

O.025 

24.66 

P < 

0.001* 

23.796 

0.002* 

Relevance 

2.71 

0-439 

0.517 

0.472 

0.216 

0.642 

2-73 

0.099 

5-679 

0.683 

Clarity 

5-651 

0.13 

0.001 

0.978 

3-977 

0.046 

9.26 

0.002* 

45.702 

P< 

0.001* 

Consistency 

5-564 

0.135 

0.942 

0.322 

4.04 

0.044 

5-39 

0.02 

21.154 

0.007 

Modularity 

8.662 

0.034 

2.047 

0.153 

0.378 

0-539 

0.895 

0-344 

17.942 

0.022 
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Organization 

0.596 

0.897 

0.006 

0-937 

0.164 

0.685 

7.19 

0.007 

22.14 

0.005* 

Interface 

0-545 

0.909 

0 

0.986 

0-397 

0.528 

2.68 

0.102 

25-584 

0.001* 

Grammar 

7-595 

0.055 

2.285 

0.131 

4-393 

0.036 

2.19 

0.138 

7.487 

0.485 


Cultural relevance 

5-562 

0-135 

0-155 

0.1694 

0.278 

0.598 

0.675 

0.411 

15-934 0.043 

Overall quality 

3-809 

0.283 

0.275 

0.6 

1.582 

0.208 

13.746 

P < 

0.001* 

19.771 0.011 


“Bonferroni correction for eleven tests required p-values to be less than 0.0045. 


Table 3 

Ratings by Reviewer Status 


Reviewer title 

Assistant 
Professor 
(Non-tenured, 
tenure track) 

N = 56 

Associate or Full 

Professor 

(Tenured) 

N = 107 

Adjunct 

(Non-tenure 

track) 

N = 220 

Other 

(Non-tenure 

track) 

N = 33 

Total 

N = 416 



Mean 

Median 

Mean 

Median 

Mean 

Median 

Mean 

Median 

Mean 

Median 

Comprehensiveness 

4.18 

4.00 

4.18 

4.00 

4.28 

4.00 

4.12 

4.00 

4-23 

4.00 

Accuracy 

4-54 

5-00 

4.29 

4.00 

4-45 

5-00 

4.27 

4.00 

4.41 

5-00 

Relevance & 
longevity 

4-25 

500 

4.21 

4.00 

4-35 

4.00 

4.12 

4.00 

4.28 

4.00 

Clarity 

4-55 

500 

4-33 

5-00 

4.46 

5-00 

4-36 

4.00 

4-43 

5-oo 

Consistency 

4-73 

5-oo 

4-54 

5-oo 

4-55 

5-00 

4.48 

5-00 

4-57 

5-oo 

Modularity 

4-45 

5-oo 

4-38 

5-oo 

4-59 

5-00 

4.24 

4.00 

4-49 

5-oo 

Organization & flow 

4-25 

4.00 

4-30 

4.00 

4.26 

4.00 

4-36 

500 

4.28 

4.00 

Interface 

3-95 

4.00 

4.04 

4.00 

4-05 

4.00 

4.12 

4.00 

4-04 

4.00 

Grammar 

4-77 

5-oo 

4-55 

5-oo 

4-77 

5-00 

4.76 

500 

4.71 

5-oo 

Cultural relevance 

4.20 

5-oo 

4.11 

4.00 

4.29 

5-00 

3-91 

4.00 

4.20 

4.00 

Overall quality 

4-38 

4-50 

4.29 

4.40 

4.40 

4-50 

4.27 

4-30 

4-36 

4-50 


As seen in Table 4, US and Canadian reviewers generally rated the textbooks favorably. There were some 
significant differences between US and Canadian reviewers (Table 2). These occurred on 
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Comprehensiveness, Accuracy, Clarity, and Overall Quality. In all these cases, the Canadian reviewers 
tended to rate the textbooks lower than the US reviewers. 

Table 4 

Ratings by Reviewer University 


Criteria 

UMN (U.S.) 

N = 372 

Mean Median 

BCC (Canada) 

N =44 

Mean Median 

Total 

N=4i6 

Mean 

Median 

Comprehensive¬ 

ness* 

4.27 

4.00 

3-82 

4.00 

4-23 

4.00 

Accuracy* 

4-47 

500 

3-89 

4.00 

4.41 

5-oo 

Relevance & 
longevity 

4-31 

4.00 

4-05 

4.00 

4.28 

4.00 

Clarity* 

4.48 

500 

4-05 

4.00 

4-43 

5-oo 

Consistency 

4-59 

500 

4-36 

500 

4-57 

5-oo 

Modularity 

4-50 

500 

4-39 

500 

4-49 

5-oo 

Organization & 
flow 

4-31 

500 

4.00 

4.00 

4.28 

4.00 

Interface 

4.07 

4.00 

3-82 

4.00 

4.04 

4.00 

Grammar 

4.72 

500 

4.61 

500 

4.71 

5-oo 

Cultural 

relevance 

4.22 

4.00 

4-05 

4.00 

4.20 

4.00 

Overall quality* 

4-39 

4-50 

4.10 

4.00 

4-36 

4-50 


^Denotes significant differences, with p < 0.0045 


There were some significant differences in ratings among the textbook types (Table 5). These occurred on 
Accuracy, Clarity, Organization, and Interface. In the post-hoc pairwise comparisons the 12 
Physics/Chemistry textbooks tended to be the consistently outlying (lower-rated) textbooks. 
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Table 5 

Mean Ratings by Text Type 


Criterion 

Math 

& 

Stats 

Biology & 
Physiology 

Physics & 
Chemistry 

Social 

Science 

English/ 

Humanities/ 

Communication 

Computer 

Sci/IS 

Business 
& Econ 

Law 

Other 

Total 

Compre¬ 

hensiveness 

4.28 

4-36 

3-87 

4-3 

4.14 

4.2 

4.24 

443 

4.17 

4-23 

Accuracy* 

4-59 

4.2 

3-97 

4.42 

444 

473 

443 

471 

4-38 

441 

Relevance & 
longevity 

4.41 

4-38 

4-1 

4.19 

4.29 

4.2 

4.24 

4-57 

4.28 

4.28 

Clarity* 

4-31 

4.07 

4 

472 

4-63 

44 

4-32 

5 

448 

443 

Consistency 

4.66 

4-35 

4.29 

4.66 

4-59 

4.67 

4.62 

5 

4-52 

4-57 

Modularity 

4.42 

4-53 

3-97 

4.66 

4-54 

4.67 

443 

4.29 

4-52 

449 

Organization* 

&flow 

4-45 

4.29 

3-68 

4-39 

4.11 

4.27 

4.27 

4.86 

448 

4.28 

Interface* 

4-3 

4.02 

3-52 

4.27 

3-86 

4.27 

3-68 

371 

4.21 

4.04 

Grammar 

47 

475 

4.42 

473 

477 

4.67 

4-65 

5 

479 

471 

Cultural 

relevance 

4.28 

4-25 

4-13 

4.26 

4.01 

4.6 

4.24 

4-57 

3-93 

4.2 

Overall 

quality 

4-44 

4-32 

3-99 

4.46 

4-34 

447 

4-31 

4.61 

4-38 

4-36 


^Denotes significant differences, with p < 0.0045 


Discussion 

Our first research question was, “What are the average ratings of each quality criterion and overall quality?” 
In answer to this question, we found that textbook quality was generally rated high for each criterion and 
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for overall quality. The highest-rated criteria were grammar (4.67) and consistency (4.52). The lowest-rated 
criteria were interface (3.95) and comprehensiveness (4.13). This finding indicates that reviewers are 
identifying strengths and weaknesses of the textbooks they review across the different categories. The fact 
that the reviews were favorable across all nine criteria indicates that the reviewers judged the textbooks 
housed on the Open Textbook Library to be of relatively high quality. 

Our second and third research questions asked whether reviewers’ professional track and status affect their 
ratings. We found no significant difference in the quality of ratings by tenure-track and non-tenure track 
reviewers. Dividing reviewers into four groups—Adjunct, Tenure Track Assistant Professors, Tenured 
Associate or Full Professors, and Other (such as student life advisors that teach courses)—again yielded no 
significant difference in the quality of reviews. This finding has important implications for research studies 
of OER use at community colleges, where many instructors are adjuncts. The data suggests that quality can 
be identified without particular bias according to status; more experienced teachers (such as full professors) 
were not more discriminating than less-experienced teachers in terms of the quality of textbooks. In other 
words, in studies involving faculty ratings, it would not be accurate to argue that reviews should be 
discounted because they are by faculty of different ranks, because faculty in this study tended to agree about 
the textbook quality, regardless of rank. 

Fourth, we asked whether U.S. and Canadian reviewers rate the textbooks differently. Across all categories, 
typical ratings by Canadian reviewers were lower than typical ratings by U.S. raters. Our data indicates that 
raters’ countiy of residence predicts some differences in ratings; however, with the data we have we cannot 
determine why this occurred. We might speculate that the reviewers from Canada seem to have higher 
standards than U.S. reviewers. There are a few important implications of this finding. First, although OER 
are typically available online facilitating international use, they may not be perceived as being equally useful 
in different educational systems. In some contexts, instructors may need to significantly revise open 
textbooks produced elsewhere, which revising is allowed under most open licenses. Second, when 
consulting online reviews of open textbooks, readers should be aware of reviewers’ contexts. Reviews may 
be most useful when the reviewer and reader teach similar courses in the same countiy. Third, in studies 
such as this one involving raters from multiple countries, average scores may hide differences among 
various groups of raters. 

Finally, we asked whether quality varies among textbook types. According to the data, perceived quality 
varies by discipline or textbook type. Among the nine disciplinary categories, physics and chemistry books 
tended to be rated significantly lower in some criteria than textbooks in the other disciplines. In our data, 
there is a correlation between low scores for science textbooks and low ratings by Canadian reviewers; 
Canadian reviewers rated a higher proportion of physics and chemistry books than any other disciplinary 
categoiy. 

Taken together, the data indicates that overall quality of textbooks in the Open Text Library is high, but 
textbooks do vary in quality, and those variances can be identified by reviewers. Therefore, online ratings 
can be used to help consumers discriminate among various open textbooks. Furthermore, quality is an 
important covariate in OER research. Up to this point, most efficacy studies involving open textbooks have 
not considered the quality of the textbooks as a covariate. In comparing the effectiveness of open textbooks 
and proprietary materials, researchers need to acknowledge and account for differences in quality among 
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available texts, whether open or proprietary. It is not just the presence or absence of OER at play, but the 
quality of each resource used that affects perceptions and efficacy. 


Limitations 

As noted previously, this study was limited by the available data. This study included 416 reviews of 121 
textbooks, but some textbooks received many more reviews than others. Reviewers were given veiy little 
guidance in the rating process. Of the 416 reviewers, only 44 were Canadian, and the Canadians reviewed 
only 5 of the 10 text types. Based on these limitations, the results suggest patterns, but may not be 
generalizable. Further studies are needed to understand how textbook type and reviewer characteristics 
affect textbook ratings. 

Another limitation stems from the nature of the reviews. In the Open Textbook Library, reviewers attach 
their names to their ratings. The rationale for publishing reviewers’ names was to increase rater 
accountability and consumer trust. However, it is possible that named reviewers inflate scores compared 
with anonymous reviewers. Based on this limitation, comparisons between Open Textbook Library ratings 
and anonymous ratings may not be valid. For this study, all comparisons are among Open Textbook Library 
ratings. 


Conclusion 

In the brief history of the open textbooks movement, proponents of open textbooks have necessarily focused 
much effort on development and awareness. There are now thousands of open textbooks online and 
thousands of instructors using open textbooks (MERLOT, 2015), but as yet, relatively few studies about 
perceived quality and measured effectiveness of open textbooks. This study is among the first to examine 
online textbook rating systems. We hope that others will pursue similar research to help explain OER 
ratings and quality. 

In this study, we examined faculty ratings of textbooks in the Open Textbook Library, and found that the 
ratings tend to be high, with some variance. Based on the ratings, instructors wishing to find high quality 
open textbooks should be able to trust the quality of texts in the Open Textbook Library. As Pitt (2015) 
suggested, recommendations by faculty are highly influential. However, the ratings reflect only perceptions, 
not outcomes for students who use the textbooks, nor comparisons of those outcomes with outcomes for 
students using proprietary textbooks. Perhaps the Open Textbook Network and other open textbook 
repositories could add measures of effectiveness to their online rating systems. Combining ratings with 
efficacy measures would provide a more complete picture than ratings alone, greatly benefiting potential 
users of open textbooks. 
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